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ABSTRACT 

This  paper  presents  the  past  and  current  activities  on  which  Oshkosh  Truck  Corp.  and  University  of 
Parma ’s  VisLab  are  jointly  involved  in  the  field  of  Unmanned  Ground  Vehicles.  Oshkosh  Truck  Corp.  and 
VisLab  share  a  common  vision  on  the  promise  of  autonomous  technology  in  the  future  of  ground  vehicles. 

The  paper  describes  the  activities  of  each  partner  and  shows  some  results  of  the  cooperation,  together 
with  plans  for  the  future  which  again  sees  Oshkosh  and  VisLab  joining  forces  to  compete  in  the  next 
DARPA  Urban  Challenge. 

1.0  OVERVIEW 

Unmanned  Ground  Vehicle  Technology 

Unmanned  Ground  Vehicles  (UGVs)  represent  new  technology,  with  a  history  that  scarcely  covers  more 
than  20-25  years.  Some  initial  ideas  and  concepts  were  born  in  the  1960s,  but  the  relatively  immature 
technology  of  that  early  time  could  not  meet  the  original  goal  of  fully  autonomous  all-terrain  all-weather 
vehicles.  Documented  prototypes  of  automated  vehicles  did  not  appear  until  much  later,  with  a  few 
groups  developing  functioning  systems  in  the  mid-1980s.  The  initial  stimulus  for  these  pioneering  groups 
came  from  the  military  sector,  where  leaders  envisioned  a  fleet  of  ground  vehicles  operating  with 
complete  autonomy. 

After  some  of  these  early  military  prototypes,  interest  transferred  to  the  civil  sector.  Governments 
worldwide  launched  several  pilot  projects  that  generated  interest  in  and  attracted  a  large  number  of 
researchers  to  these  topics.  Interest  of  the  automotive  industry  in  developing  real  products — and  thus  in 
investing  real  money  in  this  new  business — developed  only  after  the  government  feasibility  studies  had 
successfully  demonstrated  the  first  prototypes.  These  early  tests  of  autonomous  vehicles  on  real  roads  in  a 
real  environment  represent  one  of  the  most  important  milestones  in  the  history  of  unmanned  ground 
vehicles.  These  tests  happened  in  the  mid-to-late  1990s. 

Due  to  many  technological  and  legal  issues,  the  automotive  industry  did  not  set  full  autonomy  as  a  goal. 
Instead  of  automotive  pilots,  industry  focused  on  equipping  vehicles  with  supervised  systems  and  ADAS 
(Advanced  Driving  Assistance  Systems).  Research  on  UGVs  then  slowed  down,  with  neither  industry  nor 
government  deeming  it  as  a  primary  strategic  area  for  investment.  Departments  of  Transportation 
worldwide  were  in  fact  primarily  interested  in  social,  economic,  or  environmental  objectives  aimed  at 
enhancing  the  efficiency  of  fuel,  road  network,  and  improving  the  quality  of  life  in  terms  of  mobility. 
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However,  the  excellent  results  recently  achieved  by  automotive  ADAS  have  induced  the  military  sector  to 
give  a  new  vigorous  push  to  the  original  idea  of  automating  its  fleet  of  ground  vehicles.  The  Defense 
Advanced  Research  Projects  Agency  (DARPA)  Grand  Challenge  proved  to  be  a  very  inspired  concept.  It 
attracted  a  large  number  of  top-level  research  institutes  to  work  with  the  million-dollar  prize  in  mind  and 
in  doing  so  it  helped  the  technical  community  take  a  considerable  step  forward. 

The  DARPA  Grand  Challenge  catalyzed  the  joint  efforts  between  Oshkosh  Truck  Corporation  and  the 
University  of  Parma’s  VisLab.  The  common  interests  shared  by  these  two  important  players  in  the  field  of 
vehicles  (Oshkosh)  and  environmental  perception  (VisLab)  brought  them  together  with  the  aim  of 
participating  and  winning  the  Grand  Challenge. 

Of  course,  the  previous  description  of  UGV  applications  represents  only  a  brief  overview.  Numerous 
other  applications  could  benefit  from  unmanned  vehicle  technology  and  investigations  into  UGVs  proceed 
on  many  fronts.  The  automation  of  road  vehicles  perhaps  garners  the  most  attention  from  the  media  and 
the  most  interest  from  the  automotive  industry.  However,  other  domains  such  as  agriculture,  mine 
removal,  rescue  and  dangerous  applications  in  general  look  for  UGV  technology  to  help  manage  ever- 
increasing  personnel  costs.  If  a  vehicle  could  move  autonomously  on  a  field  to  seed,  enter  in  a  mined  field, 
or  perform  some  other  dangerous  missions,  the  efficiency  of  the  vehicle  itself  would  be  increased  thanks 
to  a  24/7  operational  schedule.  More  importantly,  many  of  these  uses  would  drastically  decrease  the 
number  of  individuals  put  at  risk. 

This  paper  focuses  on  the  design  and  development  of  TerraMax™,  a  UGV  system  that  was  fielded  to 
participate  at  the  DARPA  Grand  Challenge.  Not  only  did  TerraMax™  successfully  complete  the 
qualification  to  the  race,  but  it  concluded  the  whole  course  as  well. 

Environmental  Perception 

In  the  late  1980s  and  early  1990s,  the  scientific  community  started  to  research,  develop  and  generally 
explore  the  field  of  Intelligent  Transportation  Systems  (ITS).  Previous  military  research  into  both  ground 
and  aerial  autonomy  had  sparked  the  interest  of  academic  and  private  research  centers,  inducing  them  to 
join  together  in  a  precompetitive  research  stage. 

Autonomous  systems  pushed  the  limits  of  computing  technology  in  that  day.  Most  applications  required 
that  research  groups  develop  custom  architectures.  Some  built  these  architectures  from  off-the-shelf 
components;  others  built  their  systems  from  scratch,  beginning  with  silicon.  However,  they  all  needed  to 
be  specialized  to  match  the  specific  needs  of  the  autonomous  system,  and  even  with  that  customization  the 
computing  systems  often  proved  the  limiting  factor. 

Researchers  commonly  opted  for  sensing  technologies  such  as  radar  or  acoustic  sensors  that  could  directly 
measure  quantities  of  interest.  The  small  amount  of  data  fed  by  these  devices  did  not  require  high 
computational  capabilities.  Conversely,  vision  sensors  demanded  a  complex  processing  phase.  However, 
while  the  computational  requirements  of  vision  posed  many  challenges,  vision  offered  a  rich  and  full 
description  even  in  unstructured  conditions.  This  great  potential  led  researchers  to  continue  development 
of  systems  that  could  handle  the  computational  requirements  of  vision. 

Initial  results — despite  being  at  an  embryonic  stage — demonstrated  the  viability  of  vision  for  intelligent 
vehicles.  Dedicated  hardware  allowed  vision  based  detection  systems  to  run  in  real-time.  These  very 
early  systems  processed  only  grey-level  images.  Color  images  required  processing  of  larger  amounts  of 
data,  leading  to  unaffordable  computational  costs. 

Results  of  the  precompetitive  phase  showed  great  promise,  inspiring  pioneering  projects  at  a  few  research 
centers.  Concept  development  eventually  transformed  vehicles  into  bona  fide  intelligent  vehicle 
prototypes.  In  this  early  explorative  phase  the  scientific  community  had  come  to  appreciate  the  potential 
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of  ITS,  but  interest  from  automotive  industry  had  not  yet  coalesced  to  fund  the  research.  Instead,  research 
efforts  resulted  from  the  individual  initiative  of  single  groups  or  from  institutions  willing  to  invest  in  areas 
characterized  by  a  high  degree  of  risk.  The  three  main  examples  of  such  pioneering  projects  were  the 
VaMoRs  prototype  demonstrated  by  the  Universitat  der  Bundeswehr,  Germany  [1],  the  NavLab  vehicle 
developed  at  the  Carnegie  Mellon  University,  U.S.A.  [2],  and  the  ARGO  vehicle,  developed  by  VisLab, 
Italy  [3]. 

Technological  solutions  implemented  in  these  projects  differed  greatly.  Some  research  groups  chose 
general  purpose  hardware  (i.e.  a  more  “off-the-shelf’  solution),  some  turned  to  a  completely  customized 
processing  engine,  and  others  adopted  a  mixed  solution.  However  all  these  projects  chose  to  use  machine 
vision  as  the  main  sensing  system.  Desire  to  minimize — or  even  eliminate — any  invasion  into  the 
environment  suggests  the  use  of  passive  sensors  to  perceive  the  surroundings.  Sensor  cost  also  drives  the 
development  of  driving  assistance  systems  which  could  find  use  in  common  road  vehicles.  Use  in  mass- 
produced  passenger  vehicles  demands  that  safety  devices  become  very  inexpensive  additions. 

These  first  tests  (Munich-Odense,  No  Hands  Across  America,  and  the  MilleMiglia  in  Automatico  [1],  [2], 
[3])  and  other  similar  projects  helped  to  demonstrate  the  possibility  of  performing  vision-based 
automotive  tasks  in  real-time  with  commercial  hardware. 

Oshkosh  and  VisLab  Motivation  for  Past  Efforts  and  Vision  for  the  Future 

Unmanned  Ground  Vehicle  (UGV)  technology  plays  a  key  role  in  efforts  by  the  Oshkosh  Truck 
Corporation  (Oshkosh)  to  supply  the  American  military  forces  and  their  allies  with  the  tools  they  need. 
The  US  Congress  has  set  an  aggressive  goal,  targeting  that  one-third  of  all  operational  ground  military 
vehicles  will  be  unmanned  by  2015.  In  working  with  the  armed  forces  to  meet  this  goal,  Oshkosh 
continues  to  research  both  fully  autonomous  systems  and  leader- follower  concepts.  Autonomous  or 
leader-follower  technologies  do  not  necessarily  have  to  mean  there  are  no  soldiers  in  the  cab:  in  some 
cases  soldiers  may  still  remain  in  the  vehicle,  but  will  be  free  to  perform  tasks  other  than  driving  the 
vehicle.  Oshkosh  believes  that  while  UGVs  should  require  less  support  from  the  soldiers,  they  cannot 
provide  any  less  support  to  the  soldiers.  UGVs  need  to  do  everything  that  “conventional”  vehicles  can 
do — this  includes  operating  on  extreme  terrains,  hauling  a  load  and  even  carrying  a  crew. 

For  over  15  years,  the  Artificial  Vision  and  Intelligent  Systems  Lab  (VisLab)  of  the  University  of  Parma 
has  been  conducting  research  in  the  field  of  environmental  perception  for  both  on-  and  off-road 
applications.  Cameras  have  been  integrated  on  a  plethora  of  different  vehicle  platforms  with  the  aim  of 
sensing  the  presence  of  obstacles,  other  road  players,  potential  threats,  locating  road  features  such  as  lane 
markings  and  road  signs,  and  even  identifying  the  drivable  path  in  off-road  environments.  Together  with 
cameras,  other  sensors  have  been  included  in  the  on-board  system.  Data  fusion  techniques  have  been 
developed  and  successfully  fielded  on  many  vehicle  prototypes.  VisLab’s  ultimate  goal  is  to  apply  the 
results  of  basic  and  advanced  research  in  the  development  of  UGVs  for  unmanned  missions  in  real 
environments. 


2.0  THE  TERRAMAX™  TECHNOLOGY 

DARPA  Grand  Challenge  Platform 

To  stimulate  autonomous  vehicle  development  and  leverage  ingenuity  DARPA  issued  a  Grand  Challenge 
to  the  technical  community.  Oshkosh  and  their  partners  gave  a  big  response:  the  first  TerraMax™ 
vehicle.  Figure  1  shows  the  first  vehicle  in  Grand  Challenge  I  (2004)  configuration.  The  vehicle  was 
based  on  Oshkosh’s  Medium  Tactical  Vehicle  Replacement  (MTVR)  MK23  truck.  At  a  curb  weight  of 
13.9  tons,  an  overall  width  of  8 ’2”  and  an  overall  length  of  26 ’3,”  the  MTVR  represents  a  much  larger 
platform  than  any  other  DARPA  contestant  used.  This  extra  size  can  pose  additional  challenges  when 
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negotiating  narrow  passes  and  tunnels.  However,  Oshkosh’s  autonomous  vehicle  development  efforts 
have  focused  on  applications  that  are  logistically  relevant.  DARPA  clearly  had  logistic  military  vehicles 
in  mind  when  they  developed  the  Grand  Challenge.  Oshkosh  wanted  to  successfully  compete  in  a  vehicle 
that  could  truly  provide  the  functionality  DARPA  intended,  selecting  a  vehicle  that  could  haul  payloads  of 
15  tons  on-road  and  7.1  tons  off-road.  Vehicle  design  targeted  severe  off-road  operation:  at  the  full  off¬ 
road  payload  the  vehicle  can  still  climb  a  60%  grade  and  negotiate  a  30%  side  slope.  Features  such  as  the 
Talc-4™  independent  suspension,  central  tire  inflation  system  (CTIS)  and  all-wheel  drive  with  locking 
inter-  and  intra-axle  differentials  help  the  MTVR  meet  its  70%  off-road  mission  profile.  A  425  hp  Cat  C- 
12  engine  provides  the  power  the  vehicle  needs,  while  a  7-speed  Allison  HD4070  transmission  helps  to 
deliver  the  power  to  the  wheels  across  the  wide  range  of  speeds  the  MTVR’s  mission  demands. 


Figure  1:  The  first  TerraMax™  vehicle  in  Grand  Challenge  I  (2004)  condition 


Autonomous  Actuation 

The  autonomous  system  created  for  TerraMax™  has  been  designed,  developed  and  implemented  as  a  “kit” 
that  could  be  incorporated  on  other  logistic  vehicles.  The  autonomous  kit  must  provide  for  an  easy 
transition  between  “manual”  and  autonomous  mode.  One  purpose  of  autonomy  is  to  get  humans  out  of 
harm’s  way  and  thus  completely  out  of  the  vehicle.  However,  other  uses  keep  the  soldiers  in  the  vehicle 
with  the  autonomous  system  freeing  them  to  perform  other  tasks.  This  design  philosophy  means  that 
autonomous  actuation  cannot  prevent  or  interfere  with  a  driver  from  operating  the  vehicle  in  a 
“conventional”  fashion.  The  system  should  easily  return  “standard”  of  control  engine  and  transmission  to 
the  driver.  Moreover,  the  driver  must  always  have  overriding  manual  operation  of  safety  critical  systems 
such  as  steering  and  especially  brakes. 

As  with  most  modern  diesel  engines,  the  Cat  C-12  always  uses  “throttle”  by  wire.  The  autonomous 
system  provides  a  pulse  width  modulated  (PWM)  signal  which  could  replace  the  signal  supplied  by  the 
accelerator  pedal.  Switching  between  manual  and  autonomous  mode  is  a  matter  of  changing  between 
these  two  inputs  to  the  engine  control  unit  (ECU).  The  autonomous  transmission  control  interfaces  with 
the  transmission  control  harness,  pulling  the  outputs  from  the  driver’s  keypad  to  appropriate  levels. 
Disabling  these  extra  signals  will  return  control  to  the  driver  (via  the  keypad).  Braking  control  used  an 
electrically  actuated  pneumatic  valve,  shown  in  Figure  2,  placed  in  parallel  with  the  treadle  valve  (brake 
pedal).  The  parallel  control  circuits  keep  the  driver’s  pedal  functional  at  all  times,  and  the  use  of  the  stock 
actuation  and  braking  system  (which  has  been  tested  and  certified  to  all  safety  rules)  guarantees  that  the 
system  can  meet  federal  motor  vehicle  safety  standards  (FMVSS). 

A  servomotor  coupled  into  the  steering  wheel  shaft  so  that  the  hand  wheel  and  the  servomotor  always 
turned  together  at  a  fixed  ratio.  The  mechanical  advantage  provided  by  the  hand  wheel  renders  any  inertia 
and  friction  of  the  motor  negligible  when  the  servomotor  drive  is  disabled.  Additionally,  there  is  enough 
mechanical  advantage  to  allow  the  driver  to  override  the  servo  when  the  drive  is  enabled.  In  2004, 
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TerraMax™  competed  as  a  front-axle  steering  vehicle.  Prior  to  the  2005  Grand  Challenge,  Oshkosh 
added  steering  to  the  vehicle’s  rear  axles.  This  enhancement  leveraged  systems  used  on  another  MTVR 
variant,  the  MK3 1  tractor,  and  decreased  the  turning  radius  to  29  feet.  Mechanical  linkages  coordinate 
front  and  rear  steering.  An  additional  dwell  mechanism  holds  the  rear  wheels  straight  during  moderate 
steering  maneuvers,  while  higher  gearing  ratios  allow  the  rear  to  quickly  “catch  up”  with  the  front  steering 
during  sharp  turns. 


Figure  2:  Autonomous  control  actuation  for  brake  and  steering 


Vehicle  Computers  and  Control 

For  Grand  Challenge  I  (2004),  the  TerraMax™  computing  center  used  industrial  rack  mounted  computers 
in  vibration  damped  weatherproof  enclosures  in  the  cargo  bed.  In  the  left  hand  side  of  Figure  3  one  can  see 
the  enclosures  in  the  front  half  of  the  bed.  The  computing  system  took  up  a  considerable  amount  of  the 
cargo  space — with  a  negative  impact  on  the  hauling  capacity  of  the  vehicle.  The  2005  Grand  Challenge 
vehicle  had  an  updated  computing  center  designed  to  fit  under  the  seat.  On  the  right  side  of  side  of  Figure 
3  one  can  see  the  updated  position,  with  the  computers  in  a  modular  shock  absorbing  rack  below  (actually 
inside)  the  passenger  seat.  Mounting  inside  the  cab  frees  up  the  cargo  bed,  restoring  the  hauling  capacity; 
it  also  offers  more  protection  for  the  computers. 


Grand  Challenge  I  (2004)  Configuration 
Computing  center  in  cargo  body 


Grand  Challenge  II  (2005)  Configuration 
Computing  center  under  seat 


Figure  3:  Original  and  updated  computing  center 


Control  development  efforts  use  computer-aided  modeling  and  simulation,  with  ultimate  concept 
validation  coming  from  on-vehicle  testing.  A  detailed  dynamic  model  of  the  vehicle  was  created  in 
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ADAMS1 — capturing  the  properties  of  the  chassis  as  well  as  the  suspension,  steering  and  tires.  Offline 
simulations  help  determine  vehicle  mobility  and  the  ability  of  the  vehicle  to  maneuver  around  or  over 
various  features.  Figure  4  shows  one  such  simulation  studying  the  clearance  as  the  vehicle  traverses 
obstacles.  Results  of  these  simulations  could  be  processed  and  then  used  to  make  real-time  path  planning 
decisions.  ADAMS  simulations  also  allowed  evaluation  of  the  vehicle’s  lateral  stability,  providing 
information  about  critical  limits  that  the  real-time  controller  could  use.  In  addition,  the  real-time  control 
system  can  directly  use  a  more  basic  “bicycle  model”  as  found  in  many  vehicle  dynamics  texts,  e.g. 
Gillespie  [4]  or  Ellis  [5],  with  some  modifications  for  vehicles  with  more  than  two  axles. 


Minimum 

Underbody 

Clearance 


Time  (sec)  2005-02-07  15: 


Figure  4:  ADAMS  simulation  investigating  obstacle  clearance 


Navigation  and  Localization 

For  the  first  Grand  Challenge  (2004)  TerraMax™  used  solutions  from  several  different  suppliers:  two 
Novatel  Propak  GPS  units,  a  Crossbow  fiber-optic  gyroscope  (3  DOF  accelerations  and  angular  rates 
along  with  pitch  and  roll  angles)  and  a  Hewlett  Packard  magnetic  compass,  as  well  as  vehicle  wheel  speed 
information.  Part  of  the  vehicle  control  system  included  an  extended  Kalman  filter,  implemented  by  team 
members  at  the  Ohio  State  University,  which  used  raw  data  from  all  of  the  sources  to  estimate  vehicle 
state  (position,  pose,  linear  and  angular  velocities).  GPS  service  used  differential  corrections  from 
Omnistar’s  HP  subscription  service  (position  error  of  <=10  cm).  In  the  second  challenge  (2005)  Team 
TerraMax™  selected  a  more  integrated  approach,  based  on  GPS/INS  systems  from  Oxford  Technical 
Solutions.  The  Oxford  RT3100  includes  a  micro-electromechancical  system  (MEMS)  based  inertial 
measurement  unit  (IMU),  a  GPS  receiver  and  a  Kalman  filter  to  use  data  from  both  the  GPS  and  IMU  to 
estimate  state.  Additionally,  differential  corrections  can  be  used  for  increased  precision  and  a  wheel  speed 
signal  enhances  dead  reckoning  during  GPS  interruptions.  TerraMax™  (in  its  Grand  Challenge  II 
configuration)  used  two  RT3100  units,  one  with  WAAS  differential  GPS  corrections  enabled  and  one 
which  received  Omnistar’s  VBS  (position  error  <=50  cm)  subscription  differential  correction  service  from 
a  Trimble  receiver.  Averaging  position  solutions  from  the  two  systems  increased  accuracy  and  robustness. 
Partners  at  Rockwell  Collins  developed  an  alternate  dead  reckoning  algorithm  using  wheel  speed  and 
steering  angle.  This  alternate  algorithm  has  better  error  characteristics  than  the  MEMS  IMU  based  dead 


1  Advanced  Dynamic  Analysis  of  Mechanical  Systems,  a  rigid  body  dynamics  simulation  package  from  MSC  for  analysis  of 
complex  mechanical  systems. 
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reckoning  solution.  The  wheel  speed/steering  angle  based  solution  has  difficulties  in  some  circumstances 
(sharp  turns)  and  both  dead  reckoning  solutions  could  only  function  for  limited  periods.  However,  in  the 
open  desert  setting  of  the  Grand  Challenge,  GPS  interruptions  were  brief  and  generally  limited  to 
tunnels/underpasses  (where  limited  turning  occurs). 


3.0  SENSING  TECHNOLOGY 
System  Overview 

TerraMax™  used  a  combination  of  vision  and  laserscanning  to  detect  its  environment,  exploiting  their 
complementary  sensing  strengths.  Laserscanning  sensors  provide  very  accurate  and  precise  range 
measurements.  Laserscanners  can  provide  a  sharp  contrast  between  obstacles  and  non-obstacles.  Vision 
systems  provide  complete  data  across  the  entire  field  of  view,  not  just  across  a  scanning  line.  This  allows 
vision  to  identify  obstacles  with  limited  extent  in  one  dimension  (e.g.  street  signs  or  posts)  regardless  of 
orientation.  Vision  can  also  detect  color  contrast  (e.g.  road  markings)  in  addition  to  dimensional  contrast. 
To  meet  the  needs  of  the  Grand  Challenge,  a  trinocular  vision  system  was  developed.  The  three  cameras 
provided  three  distinct  spacings  between  the  three  different  pairs  of  cameras.  This  camera  to  camera 
spacing  is  referred  to  as  the  baseline  of  a  stereo-vision  system.  Three  distinct  baselines  allow  accurate 
stereovision  across  a  wide  range  of  viewing  distances. 

The  sensors  were  carefully  selected  to  provide  the  required  navigation  and  perception  capability.  Figure  5 
shows  the  sensor  locations  on  TerraMax™.  The  sensors  selected  for  the  DARPA  Challenge  2005  are  as 
follows: 


•  Oxford  GPS/INS 

•  Trimble  GPS 

•  Single-plane  laserscanners  (LIDAR) 

•  Multi-plane  laserscanners  (LIDAR) 

•  Forward-looking  Vision  System 

Laserscanning  Systems 

For  the  2005  Grand  Challenge  TerraMax™  used  three  laserscanning  (LIDAR)  units:  two  single  plane 
units  and  one  multi-plane  system.  (In  the  actual  competition,  the  vehicle  carried  an  additional  multi-plane 
unit  as  a  backup.)  Two  SICK  LMS-291  single  plane  laserscanners  pointed  obliquely  downward,  scanning 
an  area  near  the  vehicle.  These  served  as  the  primary  means  for  detecting  negative  obstacles  (cliffs  and 
fall-offs).  The  IBEO  ALASCA  [6]  laserscanning  unit  provides  four  scanning  planes  separated  by  roughly 
four  degrees.  One  ALASCA  unit  sat  in  the  middle  of  the  front  bumper  (with  a  spare  just  above  the 
bumper).  This  unit  looked  for  obstacles  at  a  distance;  the  four  scanning  planes  help  ensure  the  unit  could 
see  at  a  distance  of  -60m  even  as  the  vehicle  pitched  or  the  terrain  slope  changed. 
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4-planes  LIDARs 


LIDARs 


Trinocular 
Vision  System 


GPS 


Figure  5:  The  TerraMax™  sensor  suite:  the  picture  shows  the  GPS  position,  the  three  forward 
looking  cameras,  the  two  SICKs,  and  the  multi-plane  laserscanners. 


Trinocular  Vision  System 

The  vision  system  is  based  on  multi-stereoscopic  vision  (forward  looking  trinocular  system).  Figure  5 
shows  the  trinocular  vision  sensors:  three  identical  cameras  mounted  on  a  rigid  bar  above  the  front  edge 
of  the  hood.  The  asymmetrical  location  of  the  central  camera  divides  the  1.5m  between  the  left  and  right 
cameras  into  two  distinct  camera  to  camera  separation  distances  (0.5m  from  right  to  center  and  1.0m  from 
center  to  left).  A  graduated  grid  allows  precise  calibration  and  alignment  of  the  cameras.  Holding  the  roll 
and  yaw  angles  to  zero  degrees  simplifies  the  calculations,  reducing  processing  time.  The  desired  pitch 
angle  minimizes  camera  exposure  to  direct  sunlight  (framing  just  a  small  portion  above  the  horizon)  and 
brings  the  terrain  into  camera  frame  about  4  meters  from  the  vehicle. 

The  trinocular  system  sends  three  video  streams  at  10  Hz  (640x480,  color  with  Bayer  pattern)  to  the  vision 
PC  via  a  firewire  connection.  The  PC  selects  which  stereo  pair  to  use  depending  on  the  speed  of  the 
vehicle.  Since  the  baseline  of  the  stereo  vision  system  influences  the  depth  of  view,  the  large  baseline  is 
used  at  high  vehicle  speeds  so  that  a  deeper  field  of  view  is  obtained,  the  medium  one  at  medium  speeds, 
and  the  short  baseline  is  used  at  low  speeds.  This  is  one  of  the  very  few  examples  of  very  large  baseline 
stereo  systems  (1.5  m)  used  on  rough  off-road  terrain  and  delivering  a  robust  environmental  perception  at 
more  than  50  m,  regardless  of  terrain  slope. 

Design  efforts  stressed  the  need  for  a  reliable  solution:  three  non-moving  cameras  provide  a  more 
mechanically  robust  system  than  pan-tilt  solutions  seen  on  some  other  vehicles.  Additional  considerations 
reinforced  the  selection  of  three  fixed  cameras.  Vision  must  be  able  to  sense  obstacles  at  large  distances 
(exceeding  50  m  on  rough  terrain),  requiring  use  of  a  stereo  vision  system.  Furthermore,  the  baseline  had 
to  be  large  enough  to  guarantee  depth  perception  at  the  large  distances.  Systems  based  on  moving 
cameras  using  stereo  vision  with  large  baselines  face  a  number  of  mechanical  problems,  including  non- 
negligible  vibrations  for  which  processing  must  compensate.  Using  multiple  cameras  and  selectively 
choosing  the  video  streams  for  the  appropriate  stereo  pair  provided  a  very  successful  system. 
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Vision  provides  sensing  for  both  obstacle  detection  and  path  detection  (see  Figure  6  and  Figure  7). 

1.  Image  disparity  is  first  used  to  estimate  the  average  terrain  slope  in  front  of  the  vehicle  [7].  Slope 
information  is  then  used  for  both  obstacle  detection  and  path  detection.  Any  significant  deviation 
from  the  average  smooth  slope  detected  previously  is  identified  as  an  obstacle.  The  exact  location 
of  obstacles  is  then  obtained  via  stereo  triangulation  between  the  two  views  of  the  object.  A  fairly 
precise  localization  is  obtained,  but  it  can  be  further  refined  via  sensor  fusion  with  raw  data 
coming  from  the  multi-plane  laserscanner.  This  system  can  detect  thin  vertical  posts  and  fence 
poles.  It  can  also  detect  small  obstacles  [8],  but — due  to  both  the  size  and  capabilities  of  the 
vehicle  and  to  the  team  strategy — it  was  tuned  with  very  high  thresholds,  reducing  the  number  of 
false  positives  to  a  minimum.  In  other  words,  the  capability  of  detecting  small  obstacle  was 
traded  for  increased  detection  robustness.  Even  with  this  trade,  the  system  could  detect  small 
construction  cones  used  during  both  the  tests  and  the  qualification  runs.  Since  the  vehicle  can 
negotiate  60cm  steps,  obstacles  smaller  than  60  cm  need  to  be  detected  primarily  for  speed 
management  issues. 

2.  Image  disparity  is  also  used  to  compute  the  area  in  front  of  the  vehicle  which  features  a  smooth 
slope,  the  so-called  free-space.  The  free-space  is  one  of  several  features  that  are  collectively  used 
by  the  path  detection  algorithms.  Path  detection  generates  the  free-space  along  with  information 
about  shape,  similarity  in  texture  and  similarity  in  color  and  then  fuses  all  the  data  together  to 
deliver  the  location  of  any  paths  to  the  motion-planning  supervisory  controller.  Free  space  is 
obtained  using  a  standard  image  warping  [9]  in  order  to  localize  deviations  from  a  smooth  road 
surface:  Figure  7  shows  the  right  image  (a),  the  warped  left  image  (b),  and  -in  green-  the 
matching  cluster  representing  free  space  (c).  Figure  7(d)  shows  the  final  result  of  path  detection. 
This  algorithm  also  signals  the  presence  of  a  straight  segment  of  road,  in  order  to  increase  vehicle 
speed.  When  a  curved  path  is  present  (the  red  dot  in  Figure  7  shows  the  presence  of  a  non-straight 
road),  vehicle  speed  is  reduced. 
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Figure  6:  Images  showing  left  and  right  images  of  different  situations;  on  the  right  images, 
colors  show  the  presence  of  detected  obstacle:  different  colors  indicate  different  distances. 
Posts — including  even  fence  posts — and  thin  poles  are  correctly  detected. 
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Figure  7:  Image  showing  different  steps  of  path  detection:  first  the  free-space  is  determined 
then  the  path  is  localized.  Right  image  (a),  warped  left  image  (b),  free  space  in  green  (c),  the  final 

result  of  path  detection  (d). 


4.0  TERRAMAX™  PERFORMANCE 

DARPA  Grand  Challenge  I  and  II  have  provided  an  important  measuring  stick  against  which  Oshkosh  and 
VisLab  (and  a  host  of  other  participants)  have  been  able  to  chart  their  progress  in  the  unmanned  ground 
vehicle  field.  DARPA  created  the  robotic  vehicle  competition  as  an  open  challenge  intended  to  energize 
the  technical  community  and  mobilize  them  to  tackle  the  major  issues  confronting  autonomous  vehicle 
development.  For  the  timed  competition,  DARPA  designed  a  132-mile  (215  km)  off-road  desert  course 
that  each  vehicle  had  to  negotiate.  The  course  was  defined  by  an  ordered  list  of  geographic  waypoints,  a 
maximum  speed  for  each  waypoint  and  lateral  boundaries  that  could  not  be  crossed.  Vehicles  had  to 
operate  with  full  autonomy  as  they  maneuvered  around  obstacles  lining  the  desert  course. 

In  the  first  Grand  Challenge  (March  of  2004),  Team  TerraMax™  was  one  of  only  seven  teams  to 
successfully  navigate  the  preliminary  Qualifying  Inspection  and  Demonstration  (QID)  events.  However, 
like  all  other  teams,  Team  TerraMax™  quickly  realized  just  how  “Grand”  a  Challenge  DARPA  had 
proposed:  no  team  completed  10%  of  the  course.  Over  the  next  18  months,  Oshkosh,  VisLab  and  the  rest 
of  Team  TerraMax™  continued  their  development  efforts.  The  period  saw  many  redesigns  or  design 
enhancements,  several  of  which  have  been  described  in  this  report.  In  October  of  2005  TerraMax™ 
returned  with  a  much  stronger  entry.  To  the  left  of  Figure  8  one  can  see  TerraMax™  during  a  portion  of 
the  National  Qualifying  Event  (NQE),  which  it  successfully  passed  to  qualify  for  another  Grand  Challenge 
Event  (GCE).  Of  course,  Team  TerraMax™  was  not  the  only  team  hard  at  work  between  challenges  I  and 
II.  Twenty-three  teams  qualified  for  the  GCE  in  2005  and  nearly  20  of  them  completed  at  least  10%  of  the 
course.  However,  while  many  improved,  ultimately  only  five  vehicles  were  able  to  complete  the  entire 
course.  On  the  right  side  of  Figure  8  one  can  see  TerraMax™  joining  the  winning  circle  after  completing 
the  full  course.  Along  the  way  to  the  finish  line,  TerraMax™  reached  a  top  speed  of  42  mph — an 
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impressive  feat  for  a  16-ton  vehicle  and  one  which  required  the  full  range  of  its  sensing  suite.  The  vehicle 
even  spent  a  full  night  in  the  desert;  waiting  patiently  during  the  DARPA  imposed  evening  curfew  and 
restarting  to  successfully  complete  its  mission  the  following  day. 


Figure  8:  TerraMax™  at  Grand  Challenge  II  (2005):  successfully  qualifying  and  finishing 


5.0  CONTINUED  DEVELOPMENT 

Shortly  after  the  DARPA  Grand  Challenge  ended,  Oshkosh,  VisLab  and  their  partners  demonstrated  the 
autonomous  navigation  system  “kit”  on  a  much  larger  vehicle,  using  the  Palletized  Loading  System  (PLS) 
as  the  target  platform.  The  PLS  is  a  10x10  vehicle  with  a  gross  vehicle  weight  of  44  tons  and  the 
capability  of  delivering  16.5  tons  of  payload.  This  heavy  hauler  serves  as  one  of  the  US  Army’s  premier 
logistical  workhorses.  Figure  9  shows  the  autonomous  PLS  operating  off-road  with  full  load.  The  project 
took  approximately  75  days  to  complete.  In  January  of  2006  (literally  on  the  heels  of  the  October  2005 
Darpa  Grand  Challenge)  the  vehicle  was  successfully  demonstrated  at  the  US  Army’s  Yuma  test  center. 


Figure  9:  Autonomous  PLS  operating  off-road  with  full  load 

Even  with  the  aggressive  timeline,  attention  was  paid  to  the  details  of  integration.  On  the  right  side  of 
Figure  10  one  can  see  improvements  in  sensor  integration  on  the  autonomous  PLS.  Cameras  for  the  vision 
system  have  now  been  discretely  placed  inside  the  cab,  instead  of  on  a  heavy  external  support  at  the  front 
of  the  hood.  Installation  of  an  upgraded  computing  center  did  not  sacrifice  cargo  space  or  passenger 
space.  The  left  of  Figure  11  shows  the  passenger  seat  mounted  computed  center  in  TerraMax™.  The 
right  side  of  the  figure  shows  the  installation  on  the  PLS,  with  the  computers  tucked  neatly  behind  the 
passenger  seat.  Attention  to  details  such  as  this  and  efforts  to  minimize  the  intrusion  into  either  the  cargo 
space,  or  the  passenger  space  follow  from  one  of  Oshkosh’s  guiding  principles  in  autonomous  system 
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development:  vehicle  functionality  should  not  be  sacrificed  for  autonomy.  Ground  military  vehicles  are 
working  vehicles.  Autonomous  systems  that  cannot  provide  full  functionality  do  not  give  armed  forces 
the  tools  they  need. 


Figure  10:  Oshkosh  Truck  unmanned  vehicles:  TerraMax™,  based  on  MTVR,  (left)  and 

autonomous  PLS  (right) 


TerraMax™  Computing  Center 


Autonomous  PLS  Computing  Center 


Figure  1 1 :  Integration  of  computing  center  in  seat  of  TerraMax™  (left)  and  upgraded  computing 

center  in  Autonomous  PLS  (right) 


6.0  NEXT  CHALLENGE 

Oshkosh  and  the  University  of  Parma  have  again  partnered,  joining  with  Teledyne  Scientific  and  Imaging 
Sensors  LLC,  Auburn  University  and  IBEO  Automobile  Sensor  GmbH  to  compete  in  the  2007  DARPA 
Urban  Challenge.  This  competition  represents  the  next  milestone  in  the  development  and  demonstration 
of  military  unmanned  ground  vehicle  technology.  It  will  require  vehicles  to  operate  and  maneuver  in  the 
presence  of  other  moving  vehicles  in  a  mock  urban  environment.  The  competition  will  mimic  a  60-mile 
(97  km)  supply  mission  through  a  city,  with  a  6-hour  time  limit  for  completion.  Requirements  include 
obeying  traffic  laws,  safe  entry  into  traffic  flow  and  passage  through  busy  intersections. 

The  urban  environment  offers  several  new  challenges.  Driving  on  roads  with  traffic  can  require  more 
precision  than  following  an  open  trail.  However,  while  precision  is  required,  tall  buildings  could  lead  to 
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extended  operation  with  limited/impaired  GPS  service.  Moving  traffic  means  the  vehicle  must  be  able  to 
sense  not  only  obstacles  that  might  lie  in  its  path,  but  also  moving  obstacles  whose  paths  might  lie  in  its 
path.  This  greatly  increases  the  sensing  requirements.  When  dealing  with  only  stationary  objects  sensors 
can  primarily  focus  on  the  front  of  the  vehicle;  moving  traffic  requires  much  greater  sensing  capability 
behind  and  beside  the  vehicle.  Team  Oshkosh  Truck  will  meet  the  extended  localization  and  sensing 
needs  by  enhancing  previous  systems,  adding  new  systems  and  seeking  to  better  exploit  synergistic 
combinations  of  different  sensing  systems. 

IBEO  will  provide  their  latest  laserscanning  system,  including  a  fusion  algorithm  to  provide  a  single 
“view”  from  several  sensors.  The  vehicle  will  have  an  improved  trinocular  system,  additional  vision 
systems  with  different  viewing  orientations,  and  vision/laserscanner  fusion.  Using  multiple  camera  sets, 
each  specialized  for  perception  in  a  given  direction,  combined  with  laserscanner  technology  should  allow 
the  vehicle  to  successfully  negotiate  the  missions  and  sense  all  threats  the  Urban  Challenge  will  pose.  The 
complementary  nature  of  these  two  technologies  together  with  the  choice  of  highly  ruggedized  sensors, 
computing  engines,  and  vehicle  integration  will  prepare  TerraMax™  for  another  successful  competition. 
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